{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 逻辑斯蒂回归\n",
    "\n",
    "参考：https://github.com/fengdu78/lihang-code\n",
    "\n",
    "逻辑斯谛回归(LR)是经典的分类方法\n",
    "\n",
    "1．逻辑斯谛回归模型是由以下条件概率分布表示的分类模型。逻辑斯谛回归模型可以用于二类或多类分类。\n",
    "\n",
    "$$P(Y=k | x)=\\frac{\\exp \\left(w_{k} \\cdot x\\right)}{1+\\sum_{k=1}^{K-1} \\exp \\left(w_{k} \\cdot x\\right)}, \\quad k=1,2, \\cdots, K-1$$\n",
    "\n",
    "$$P(Y=K | x)=\\frac{1}{1+\\sum_{k=1}^{K-1} \\exp \\left(w_{k} \\cdot x\\right)}$$\n",
    "这里，$x$为输入特征，$w$为特征的权值。\n",
    "\n",
    "逻辑斯谛回归模型源自逻辑斯谛分布，其分布函数$F(x)$是$S$形函数。逻辑斯谛回归模型是由输入的线性函数表示的输出的对数几率模型。\n",
    "\n",
    "2．最大熵模型是由以下条件概率分布表示的分类模型。最大熵模型也可以用于二类或多类分类。\n",
    "\n",
    "$$P_{w}(y | x)=\\frac{1}{Z_{w}(x)} \\exp \\left(\\sum_{i=1}^{n} w_{i} f_{i}(x, y)\\right)$$\n",
    "$$Z_{w}(x)=\\sum_{y} \\exp \\left(\\sum_{i=1}^{n} w_{i} f_{i}(x, y)\\right)$$\n",
    "\n",
    "其中，$Z_w(x)$是规范化因子，$f_i$为特征函数，$w_i$为特征的权值。\n",
    "\n",
    "3．最大熵模型可以由最大熵原理推导得出。最大熵原理是概率模型学习或估计的一个准则。最大熵原理认为在所有可能的概率模型（分布）的集合中，熵最大的模型是最好的模型。\n",
    "\n",
    "最大熵原理应用到分类模型的学习中，有以下约束最优化问题：\n",
    "\n",
    "$$\\min -H(P)=\\sum_{x, y} \\tilde{P}(x) P(y | x) \\log P(y | x)$$\n",
    "\n",
    "$$s.t.  \\quad P\\left(f_{i}\\right)-\\tilde{P}\\left(f_{i}\\right)=0, \\quad i=1,2, \\cdots, n$$\n",
    " \n",
    " $$\\sum_{y} P(y | x)=1$$\n",
    " \n",
    "求解此最优化问题的对偶问题得到最大熵模型。\n",
    "\n",
    "4．逻辑斯谛回归模型与最大熵模型都属于对数线性模型。\n",
    "\n",
    "5．逻辑斯谛回归模型及最大熵模型学习一般采用极大似然估计，或正则化的极大似然估计。逻辑斯谛回归模型及最大熵模型学习可以形式化为无约束最优化问题。求解该最优化问题的算法有改进的迭代尺度法、梯度下降法、拟牛顿法。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "回归模型：$f(x) = \\frac{1}{1+e^{-wx}}$\n",
    "\n",
    "其中wx线性函数：$wx =w_0\\cdot x_0 + w_1\\cdot x_1 + w_2\\cdot x_2 +...+w_n\\cdot x_n,(x_0=1)$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-10-30T12:33:54.783250Z",
     "start_time": "2019-10-30T12:33:53.450260Z"
    }
   },
   "outputs": [],
   "source": [
    "from math import exp\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "from sklearn.datasets import load_iris\n",
    "from sklearn.model_selection import train_test_split"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 获取训练数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-10-30T12:33:54.794221Z",
     "start_time": "2019-10-30T12:33:54.789234Z"
    }
   },
   "outputs": [],
   "source": [
    "def create_data():\n",
    "    iris = load_iris()\n",
    "    df = pd.DataFrame(iris.data, columns=iris.feature_names)\n",
    "    df['label'] = iris.target\n",
    "    df.columns = ['sepal length', 'sepal width', 'petal length', 'petal width', 'label']\n",
    "    # 取100行数据，sepal length , sepal width 和 label 列\n",
    "    data = np.array(df.iloc[:100, [0,1,-1]])\n",
    "    # print(data)\n",
    "    return data[:,:2], data[:,-1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-10-30T12:44:11.103756Z",
     "start_time": "2019-10-30T12:44:11.098770Z"
    }
   },
   "outputs": [],
   "source": [
    "# 获取数据和标签\n",
    "X, y = create_data()\n",
    "# 使用 sklearn 的train_test_split方法，对数据进行随机划分\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-10-30T12:44:11.646492Z",
     "start_time": "2019-10-30T12:44:11.642502Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[6.1 3. ]\n",
      " [5.1 3.3]\n",
      " [4.7 3.2]\n",
      " [5.1 3.4]\n",
      " [5.7 2.8]]\n"
     ]
    }
   ],
   "source": [
    "print(X_train[:5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## LogisticRegression 模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-10-30T12:53:56.317182Z",
     "start_time": "2019-10-30T12:53:56.306211Z"
    }
   },
   "outputs": [],
   "source": [
    "class LogisticRegressionClassifier():\n",
    "    def __init__(self, max_iter=200, learning_rate=0.01):\n",
    "        self.max_iter = max_iter\n",
    "        self.learning_rate = learning_rate\n",
    "        \n",
    "    def sigmoid(seld, x):\n",
    "        return 1 / (1 + exp(-x))\n",
    "    \n",
    "    def data_matrix(self, X):\n",
    "        \"\"\"\n",
    "        给训练鼠每一行增加一列 1\n",
    "        :param X: 训练数据\n",
    "        \"\"\"\n",
    "        data_mat = []\n",
    "        for d in X:\n",
    "            data_mat.append([1.0, *d])\n",
    "        return np.array(data_mat)\n",
    "    \n",
    "    def fit(self, X, y):\n",
    "        # 增加第一列全1\n",
    "        data_mat = self.data_matrix(X)\n",
    "        self.weights = np.zeros(len(data_mat[0]), dtype=np.float32)\n",
    "        \n",
    "        for iter_ in range(self.max_iter):\n",
    "            for i in range(len(X)):\n",
    "                # 随机梯度下降\n",
    "                # 统计学习方法中 对 6.1.3 对数似然函数求导为 xi * yi - (exp(w * xi) * xi) / (1 + exp(w * xi))\n",
    "                # 似然函数最大，梯度下降变为梯度上升\n",
    "                xi = data_mat[i]\n",
    "                yi = y[i]\n",
    "                wx = np.dot(xi, self.weights)\n",
    "                self.weights += self.learning_rate * (xi * yi - (exp(wx) * xi) / (1 + exp(wx)))\n",
    "                \n",
    "                # 梯度求导方法也可以使用下面的方式, 没有做 exp 变换，求导所得\n",
    "#                 result = self.sigmoid(np.dot(data_mat[i], self.weights))\n",
    "#                 error = y[i] - result\n",
    "#                 self.weights += self.learning_rate * error * [data_mat[i]\n",
    "        print('LogisticRegression Model(learning_rate={},max_iter={})'.format(\n",
    "        self.learning_rate, self.max_iter))\n",
    "    def predict(self, test_data):\n",
    "        \"\"\"\n",
    "        根据测试集返回预测结果\n",
    "        :param test_data:\n",
    "        :return: array\n",
    "        \"\"\"\n",
    "        all_prob = self.predict_prob(test_data)\n",
    "        result_label = [1 if i > 0.5 else 0 for i in all_prob]\n",
    "        \n",
    "        return result_label\n",
    "        \n",
    "    def predict_prob(self, test_data):\n",
    "        \"\"\"\n",
    "        根据测试集返回预测概率\n",
    "        :param test_data:\n",
    "        :return: array\n",
    "        \"\"\"\n",
    "        all_prob = np.zeros(len(test_data))\n",
    "        test_data = self.data_matrix(test_data)\n",
    "        \n",
    "        for i in range(len(test_data)):\n",
    "            all_prob[i] = self.sigmoid(np.dot(self.weights, test_data[i]))\n",
    "            \n",
    "        return all_prob\n",
    "    \n",
    "    def score(self, test_data, test_label):\n",
    "        \"\"\"\n",
    "        :param test_data: 测试数据\n",
    "        :param test_label: 测试数据标签\n",
    "        :return:\n",
    "        \"\"\"\n",
    "        predict_label = self.predict(test_data)\n",
    "\n",
    "        error_cnt = 0\n",
    "        for k in range(len(predict_label)):\n",
    "            # 记录误分类数\n",
    "            if predict_label[k] != test_label[k]:\n",
    "                error_cnt += 1\n",
    "        # 正确率 = 1 - 错误分类样本数 / 总样本数\n",
    "        print('accuracy: ', 1 - error_cnt / len(predict_label))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-10-30T12:53:57.455218Z",
     "start_time": "2019-10-30T12:53:57.342484Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LogisticRegression Model(learning_rate=0.01,max_iter=200)\n"
     ]
    }
   ],
   "source": [
    "lr_clf = LogisticRegressionClassifier()\n",
    "lr_clf.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-10-30T12:53:58.155633Z",
     "start_time": "2019-10-30T12:53:58.152603Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "accuracy:  1.0\n"
     ]
    }
   ],
   "source": [
    "lr_clf.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 画图展示训练结果"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-10-30T13:02:19.575928Z",
     "start_time": "2019-10-30T13:02:19.424728Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x11d9b762f98>"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD8CAYAAACMwORRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xl8VPW9//HXhyQkYUskgEAWQhJEZZFNkaIQUIvirqjgVtTKVfFqa6vV+/PX9tLbXq2/3lYvAWVxV9SiIu5aCciugAgIohMCJGELW9iyTr6/PyYojFlOkjNzzpx8no8HjyRnDmc+3xn9cHLmPZ8RYwxKKaW8pZXTBSillLKfNnellPIgbe5KKeVB2tyVUsqDtLkrpZQHaXNXSikP0uaulFIepM1dKaU8SJu7Ukp5ULRTd9ypUyeTnp7u1N0rpVREWr169V5jTOeG9rPc3EUkClgFFBljLgu6bSLwBFBUs2mqMWZWfcdLT09n1apVVu9eKaUUICLbrOzXmDP3+4FNQIc6bn/dGHNvI46nlFIqRCxdcxeRFOBSoN6zcaWUUu5g9QXVfwAPAdX17HOtiKwTkbkiklrbDiIySURWiciq4uLixtaqlFLKogYvy4jIZcAeY8xqEcmuY7d3gTnGmHIRuQt4ARgdvJMxZgYwA2DIkCE6a1gp5YjKykoKCwspKytzupQ6xcXFkZKSQkxMTJP+vpVr7sOBK0RkLBAHdBCRl40xNx/fwRiz74T9ZwKPN6kapZQKg8LCQtq3b096ejoi4nQ5P2GMYd++fRQWFtKzZ88mHaPByzLGmEeMMSnGmHRgPLDgxMYOICLdTvjxCgIvvCqllCuVlZWRlJTkysYOICIkJSU16zeLJufcRWQKsMoYMx+4T0SuAKqA/cDEJleklFJh4NbGflxz62tUczfGLAQW1nz/+xO2PwI80qxKlPKQeV8V8cTHm9lxsJTuifE8OKY3Vw1Mdros1YLo+AGlbDbvqyIeeWs9RQdLMUDRwVIeeWs9874qavDvqpbjo48+onfv3mRlZfHYY4/Zfnxt7krZ7ImPN1Na6T9pW2mlnyc+3uxQRcpt/H4/kydP5sMPP2Tjxo3MmTOHjRs32nofjs2WUcqrdhwsbdR25X52X2b74osvyMrKIiMjA4Dx48fzzjvvcOaZZ9pVsp65K2W37onxjdqu3C0Ul9mKiopITf3xvZ4pKSkUFdl72U6bu1I2e3BMb+Jjok7aFh8TxYNjejtUkWqOUFxmM+an7+G0O72jl2WUstnxX9c1LeMNobjMlpKSQkFBwQ8/FxYW0r179yYfrzba3JUKgasGJmsz94juifEU1dLIm3OZ7eyzz+b7778nPz+f5ORkXnvtNV599dXmlPkTellGKaXqEYrLbNHR0UydOpUxY8ZwxhlncP3119OnT5/mlnryfdh6NKWU8phQXWYbO3YsY8eOtaPEWmlzV0qpBkTiZTa9LKOUUh6kzV0ppTxIm7tSSnmQNnellPIgbe5KKeVB2tyVUsoBt99+O126dKFv374hOb42d6UIDIca/tgCej78PsMfW6Cz11XITZw4kY8++ihkx9fmrlo8/XAN1aB1b8Df+8IfEwNf173R7EOOGDGCjh072lBc7bS5qxZPP1xD1WvdG/DufVBSAJjA13fvs6XBh5I2d9Xi6YdrqHp9NgUqg/5bqCwNbHcxbe6qxdMP11D1Kils3HaX0OauWjz9cA1Vr4SUxm13CW3uqsW7amAy/31NP5IT4xEgOTGe/76mX8QNilIhcsHvISbot7iY+MD2ZpgwYQLDhg1j8+bNpKSkMHv27GYdL5hOhVSKyJz6p8Kk//WBr59NCVyKSUgJNPbj25tozpw5NhRXN23uylXs/pR5pWzR//pmN/Nw0+auXON43vx4LPF43hzQBq9UI+k1d+UamjdX4WSMcbqEejW3Pm3uyjU0b67CJS4ujn379rm2wRtj2LdvH3FxcU0+hl6WUa4Rik+ZV6o2KSkpFBYWUlxc7HQpdYqLiyMlpelxS23uyjUeHNP7pGvuoHlzFRoxMTH07NnT6TJCSpu7co1Qfcq8Ui2R5eYuIlHAKqDIGHNZ0G2xwIvAYGAfcIMxZquNdaoWQvPmStmjMWfu9wObgA613HYHcMAYkyUi44HHgRtsqE8pT9Ncv/cZY1jw7R66JsTRp3tC2O7XUlpGRFKAS4FZdexyJfBCzfdzgQtERJpfnlLepXPkvc1fbXj36x1c8uRi7nhhFS8s2xrW+7d65v4P4CGgfR23JwMFAMaYKhEpAZKAvc2uUCmPqi/Xr2fvkauiqpp5XxUxfVEe+XuPktG5Lf/vurO4ckD3sNbRYHMXkcuAPcaY1SKSXddutWz7SYBURCYBkwDS0tIaUaZS3qO5fm8pq/Tz+pcFPLMojx0lZZzZrQPTbhrEmD5diWoV/gsZVs7chwNXiMhYIA7oICIvG2NuPmGfQiAVKBSRaCAB2B98IGPMDGAGwJAhQ9z57gGlwkRz/d5wuKySV1ZuZ9bifPYeKWdwj1P489X9yO7dGSevTjfY3I0xjwCPANScuf82qLEDzAd+ASwHxgELjFvf+qWUS2iuP7IdOFrBc8u28vzSfA6VVXF+r05MHjWQoT07OtrUj2tyzl1EpgCrjDHzgdnASyLiI3DGPt6m+pTyLM31R6Y9h8qYtSSfl1ds41iFn4vOPJXJo7IYkJrodGknEadOsIcMGWJWrVrlyH0rpVRjFew/xjOf5/HGqkKq/NVcflZ37s7O5PSutaXDQ0dEVhtjhjS0n75DVXnao/PWM2dlAX5jiBJhwtBU/uuqfk6XpSKIb88Rpi/MY97aIloJXDsohbtGZpLeqa3TpdVLm7vyrEfnreflFdt/+NlvzA8/a4NXDdlQVMK0hT4+3LCL2OhW3DqsB5NGZNAtITJe8NbmrjxrzsqCOrdrc1d1Wb1tP1MX+MjdXEz72GjuHpnJ7ef1pFO7WKdLaxRt7sqz/HW8nlTXdtVyGWNY4tvL1AU+Vubv55Q2MfzmotO49WfpJMTHOF1ek2hzV54VJVJrI49yQUxNuUN1teFfm3aTk+vj68ISTu0Qy6OXnsGNQ9No0zqy22NkV69UPSYMTT3pmvuJ21XLVuWv5v31O5mWm8fm3YdJ7RjPX67ux7WDk4mNjnK6PFtoc1eedfy6uqZl1HEVVdW8taaQ6Yvy2LbvGL26tOPvN5zF5f27Ex3lrU8d1Zy7UsrzSiv8zPliOzMXb2FnSRn9khOYPCqLn595Kq0cmPvSHJpzV65y08zlLM37cdzQ8MyOvHLnMAcrCi2d0+4Oh8oqeWn5Np5dks++oxWck96Rx67tz4henVwxIiCUtLmrkAtu7ABL8/Zz08zlnmzwx+e0H58Zc3xOO6ANPkz2H63g2SX5vLB8K4fLqhh5Wmcmj8rinJ4dnS4tbLS5q5ALbuwNbY90OqfdObtKypi5eAuvrtxOaaWfi/t0ZfKoLPqlhO8TkNxCm7tSNtM57eG3fd8xpi/K483VhfiN4cqauS+9Tq3r84W8T5u7UjbTOe3h8/3uw0xbmMf8r3cQJcK4ISncNSKTtKQ2TpfmOG3uKuSGZ3as9RLM8ExvXv/UOe2ht76whJxcHx99s4v4mCgm/iydO8/PoGtCnNOluYY2dxVyr9w5rEWlZXROe+h8kb+fqbk+Pv+umPZx0fz76CxuG96Tjm1bO12a62jOXSnlasYYFn1XTE6ujy+3HiCpbWtuP68ntwzrQYe4yJz70hyac1euYnfu2+rxNG8euaqrDZ9s3EVObh7ri0rolhDHHy4/k/FnpxHf2hsjAkJJm7sKObtz31aPp3nzyFTlr+bddTuYlpvH93uO0COpDY9d049rBqXQOtpbIwJCSR8pFXL15b5DeTy771eFVnmVn1dWbmPU3xby69e/ppUIT44fwGcPjGT8OWna2BtJz9xVyNmd+7Z6PM2bR4ZjFVW8ujIw92X3oXLOSkng/156JheeEXlzX9xEm7sKObtz31aPp3lzdyspreTFZVt5dmk+B45Vcm5GR/523QCGZyV5fu5LOOjvOSrkHhzTm/iYk18Aa07u2+rx7L5fZY+9R8p5/KNvGf7YAv726XcMSE3kzbuH8dqkYZzXAgZ6hYueuauQszv3bfV4mjd3l50lpTyzaAuvfbmd8qpqxvbtxj2jMunTveXNfQkHzbkrpUJq696jPL0ojzfXFFJt4KoBydydnUlWl3ZOlxaRNOfuEU7ltDVHrppr867D5OT6eG/dDqKjWjH+7DQmjcggtaPOfQkHbe4u5lROW3PkqjnWFhwkJ9fHpxt306Z1FL88P4NfnteTLh107ks4aXN3Mafmglu9X51bro4zxrBiy35ycn0s8e0lIT6G+y/oxcSfpXOKzn1xhDZ3F3Mqp605cmWVMYaFm4uZmutj9bYDdGoXy8OXnM7N5/agXay2Fyfpo+9iTuW0NUeuGuKvNny0YRc5uT427jxEcmI8U67sw/VDUomL0bkvbqA5dxdzKqetOXJVl0p/NXNXF3LR3xcx+dU1lFb6+eu4/uT+Nptbh6VrY3cRPXN3Mady2pojV8HKKv38c1UBTy/aQtHBUk7v2p7/nTCQsf26EaUjAlypwZy7iMQBnwOxBP4xmGuM+UPQPhOBJ4Cimk1TjTGz6juu5tyVcr8j5VW8unIbMxfnU3y4nIFpidw7KovRp3fRd5I6xM6cezkw2hhzRERigCUi8qExZkXQfq8bY+5tSrEqcj06bz1zVhbgN4YoESYMTeW/rurX5P3cnutvKQ4eq+D5ZVt5bulWSkorGZ6VxJM3DGBYps59iRQNNncTOLU/UvNjTM0fZ97Wqlzl0XnreXnF9h9+9hvzw88nNm6r+7k9198SFB8uZ9aSLby8fBtHK/xceEYX7hmVxaC0U5wuTTWSpRdURSRKRNYCe4BPjTEra9ntWhFZJyJzRSTV1iqVK81ZWWBpu9X9nJq/rnPfA/+g/f6dDZz3+AJmfL6FUad34cP7z2fWL87Wxh6hLL2gaozxAwNEJBF4W0T6GmM2nLDLu8AcY0y5iNwFvACMDj6OiEwCJgGkpaU1u3jlLH8dr9cEb7e6n9tz/V60pfgI0xfm8fZXgZfLrhmUzF0jM8norHNfIl2j0jLGmIMishC4GNhwwvZ9J+w2E3i8jr8/A5gBgRdUG1uscpcokVobd1TQNVmr+7k91+8lG3ccImehjw/W76R1VCtuGprGpJGZJHt4zS1Ng5dlRKRzzRk7IhIPXAh8G7RPtxN+vALYZGeRyp0mDK396lvwdqv7uT3X7wVrth/gjue/ZOxTi1m0uZh/G5HJkt+N5j+v7KuN3WOsnLl3A14QkSgC/xi8YYx5T0SmAKuMMfOB+0TkCqAK2A9MDFXByj2OvxjaUArG6n5uz/VHKmMMy/L2kZPrY1nePhLbxPDARafxi2HpJLSJcbo8FSI6z10pjzLG8NmmPUzN9bG24CCd28cy6fwMbhyaRlud+xKxdJ67R9idv7aaN7f7eE7Nh7d7va637g38//oTHxxIJsdcy7dV3UlOjOdPV/XlusEpOh6gBdHm7mJ256+t5s3tPp5T8+HtXq/bVXz1BvPeeYPp5feTb7qRKUX8LW4WV1w8gZgBPwmvKY/TwWEuZnf+2mre3O7jWV2H29frVmWVfp5fmk/2Pyt5qOw22lDGtJh/8Enrh7iWBcTkTnG6ROUAPXN3Mbvz11bz5nYfz6n58Hav120Ol1Xy8ortzF6yhb1HKhgiu/lzzDNkt/qak1KmJYWO1aico83dxezOX1vNm9t9PKfmw9u9Xrc4cLSC55Zt5fml+Rwqq+L8Xp2YPCqLofN+hxyq5beShJTwF6kcp5dlXMzu/LXVvLndx3NqPrzd63XankNl/Pn9jQx/fAFPffY952Yk8c7k4bx0x1DOzUhCLvw9xAT9QxgTDxf83pmClaP0zN3F7M5fW82b2308p+bD271epxTsP8Yzn+fxxqpCqvzVXH5Wd+7JzqJ31/Yn79j/+sDXz6YELsUkpAQa+/HtqkXRnLtSLuXbc4RpC328s3YHrQTGDU7h30Zkkt6prdOlKQdpzl2FhVP5dS/bUFTCtIU+Ptywi9joVtw6rAeTRmTQLcHF4wHWvaG/MbiMNnfVZE7l171q1db9TM31sXBzMe1jo7knO5Pbh/ckqV2s06XVb90b8O59UFnzYnhJQeBn0AbvIG3uqsnqy6Wf2LSt7tcSGWNY4tvL1AU+Vubv55Q2Mfz256dxy7B0EuIjZO7LZ1N+bOzHVZYGtmtzd4w2d9VkTuXXvaC62vDppt1My/XxdWEJp3aI5dFLz+DGoWm0aR1h/1vWlaPXfL2jIuy/IuUmTuXXI1mVv5r31+9kWm4em3cfJq1jG/5ydT+uHZxMbHSEzn1JSAlciqltu3KM5txVkzmVX49E5VV+5nyxnQv+ZxH3v7aWamP4xw0DWPCbkdw4NC1yGzsEXjzVfL3r6Jm7ajKn8uuRpLQi0NRnfL6FXYfK6JecwNM3D+bnZ55Kq1aR/U7ZH2i+3pU0565UCBwqq+Sl5duYvSSf/UcrOKdnR+4dlcX5vTohET7+QDlLc+4OsTvPbfV4Ts0t1/z6yfYfreDZJfm8sHwrh8uqGHlaZ+4dncXZ6R2bd2Av5ci9tBYrHFqvNncb2Z3ntno8p+aWa379R7tKypi5eAuvrtxOWZWfi/t05Z7sLPqlJDT/4F7KkXtpLVY4uF59QdVGds8jt3o8p+aW273eSLR93zEeeWs9I/6ay/PLtnJJ36588qsRTL95sD2NHerPkUcaL63FCgfXq2fuNrI7z231eE7NLW/J+fXvdh9m+sI85n+9gygRrhsSmPuSltTG/jvzUo7cS2uxwsH1anO3kd15bqvHc2pueUvMr68vLGFq7vd8/M1u4mOiuO1n6dw5IoNTO8SF7k69lCP30lqscHC9elnGRnbnua0ez6m55S0pv/5F/n5uffYLLp+6hGV5+/j30VksfXg0j152ZmgbO3grR+6ltVjh4Hr1zN1Gdue5rR7PqbnlXs+vG2NY9F0xObk+vtx6gKS2rXno4t7ccm4P2seFce6Ll3LkXlqLFQ6uV3PuSgWprjZ8snEXObl5rC8qoVtCHJNGZDD+7DTiW0fwO0mVJ2jO3SNaWm7eSVX+auZ/vYNpC/Pw7TlCelIbHr+2H1cPTKF1tF7BjHjvPQCrnwfjB4mCwRPhsv9p+vFcntfX5u5iLS0375TyKj9zVxfy9KI8CvaX0vvU9jw5fgCX9utGdJQ2dU947wFYNfvHn43/x5+b0uAjIK+v/+W6WEvLzYfbsYoqZi3ewoi/5vJ/3t5Ax7axzLx1CB/efz5XDkjWxu4lq59v3PaGREBeX8/cXayl5ebDpaS0kheXbeXZpfkcOFbJuRkd+dt1AxielaRzX7zK+Bu3vSERkNfX5u5iLS03H2p7j5Qze0k+Ly3fxpHyKkaf3oXJozIZ3KOZc1+U+0lU7Y1cmvgCeQTk9fX3Thdrabn5UNlZUsof53/DeY8v4OlFeYw8rTPv33cez048Wxt7SzF4YuO2NyQC8vp65u5iLS03b7ete4/y9KI83lxTiDGB9d+dnUlm53ZOl6bC7fiLpnalZSIgr99gzl1E4oDPgVgC/xjMNcb8IWifWOBFYDCwD7jBGLO1vuNqzl2FyuZdh8nJ9fHeuh1ER7Vi/NmpTBqRQcopIZj7olSY2ZlzLwdGG2OOiEgMsEREPjTGrDhhnzuAA8aYLBEZDzwO3NCkyl3Kaj7c7fPNrebXI3G9awsOkpPr49ONu2nbOoo7z8/gjqQNdFk2Hr6y6ezKalba7gy024/nJKtr8dKaLWiwuZvAqf2Rmh9jav4En+5fCfyx5vu5wFQREePU219tZjUf7vb55lbz65G0XmMMK7bsJyfXxxLfXhLiY7j/gl7cNjydRN88e7PIVrPSdmeg3X48J1ldi5fWbJGlF1RFJEpE1gJ7gE+NMSuDdkkGCgCMMVVACZBkZ6FOspoPd/t8c6v59UhYrzGG3G/3MO7p5UyYuYJvdx3mkUtOZ+nDo/n1RaeR2Ka1/Vlkq1lpu+/X7cdzktW1eGnNFll6QdUY4wcGiEgi8LaI9DXGbDhhl9oycj85axeRScAkgLS0tCaU6wyr+XC3zze3ml9383r91YaPNuwiJ9fHxp2HSE6MZ8qVfbh+SCpxQUkg27PIVrPSdt+v24/nJKtr8dKaLWpUFNIYcxBYCFwcdFMhkAogItFAArC/lr8/wxgzxBgzpHPnzk0q2Al15cqDt1vdzyl15dSDt7txvZX+auauLuSivy9i8qtrKKv088S4/ix8MJtbh6X/tLFD3ZnjpmaR68pEB2+3+37dfjwnWV2Ll9ZsUYPNXUQ615yxIyLxwIXAt0G7zQd+UfP9OGCBV663g/V8uNvnm1vNr7tpvWWVfl5avpXsJxby239+TWx0FFNvHMinD4zkuiGpxNQ3IsDuLLLVrLTd9+v24znJ6lq8tGaLrFyW6Qa8ICJRBP4xeMMY856ITAFWGWPmA7OBl0TER+CMfXzIKnaA1Xy42+ebW82vu2G9R8ureGXlNmYuzqf4cDkD0xKZcmUfRp/exfqIALuzyFaz0nbfr9uP5ySra/HSmi3See7KVQ4eq+CFZdt4blk+B49VMjwricmjshiWoXNflAKd5+4YN+W+I0nx4XJmLdnCy8u3cbTCz4VnnMrkUZkMTDvF6dJO5vZMtd31hWIdmtkPC23uNnJD7jvSFB0sZcaiPF77soBKfzWX9u/OPdmZnNGtg9Ol/ZTbM9V21xeKdWhmP2z0soyNhj+2oNapi8mJ8Sx9eLQDFbnXluIjTF+Yx9tfFSEC1wxM4a7sTHp2aut0aXX7e986JgGmwq83NH4/t9cXinXYfUynHmsH6WUZB7g95+4Gm3YeIifXxwfrdxIT1Yqbz+3BpBEZromL1svtmWq76wvFOjSzHzba3G1k9/x1L1mz/QA5C3x89u0e2sVGM2lEJnec15PO7WOdLs06qzO8nZr1bXd9oViH3ceMgLnqTtF57jZye8493IwxLPXt5caZK7hm2jJWbz/AAxedxtLfjebhS06PrMYO7s9U211fKNahmf2w0TN3G7k95x4uxhg+27SHqbk+1hYcpEv7WB699AwmnJNG29gI/k/O7Zlqu+sLxTo0sx82+oKqso2/2vDB+p3k5Pr4dtdhUk6J566RmYwbnFL7eAClVKPpC6oqbCqqqpn3VRHTF+WRv/comZ3b8rfrzuKKAd3rHw/gZVbnvjvF7fWB+98r4HLa3FWTlVX6ef3LAmZ8voWig6X06d6B6TcNYkyfrrRq1YLfTWp17rtT3F4fuP+9AhFAL8uoRjtcVsnLK7Yze8kW9h6pYEiPU5g8Oovs0zrriACA/+xY+3hgiYI//GRYavi5vT5w/3sFHKSXZZTtDhyt4LllW3l+aT6Hyqo4v1cn7h2VxdAMz3wuiz2szn13itvrA/e/VyACaHNXDdpzqIxZS/J5ecU2jlX4GdPnVO7JzuKs1ESnS3Mniar7zNgN3F4fuP+9AhGghb7apawo2H+MR+et57y/5jJr8RZ+fuapfPLrETxzyxBt7PWxOvfdKW6vD9z/XoEIoGfu6id8e44wbaGPd9buoJXAuMEp3DUykx5JLp774iZW5747xe31gfvfKxAB9AVV9YMNRSVMW+jjww27iI1uxY3n9ODOET3plqDjE5RyC31BVVm2aut+pub6WLi5mPax0dyTncntw3uS1M6B8QBuzxjbXZ/deXO3P34qbLS5t1DGGJb49jJ1gY+V+fvp2LY1D47pzS3DetAhLsaZotyeMba7Prvz5m5//FRY6WWZFqa62vCvTbvJyfXxdWEJXTvEceeIDCack0qb1g7/W+/2jLHd9dmdN3f746dsoZdl1Emq/NW8v34n03Lz2Lz7MGkd2/Df1/TjmkHJxEa7JALn9oyx3fXZnTd3++Onwkqbu8eVV/l5a00RTy/KY9u+Y/Tq0o5/3DCAy/p3I9ptc1/cnjG2uz678+Zuf/xUWLns/25ll9IKP88uyWfkXxfyyFvrSYiP4ZlbBvPxr0Zw1cBk9zV2cH/G2O767M6bu/3xU2GlZ+4ec6iskpeWb+PZJfnsO1rBOT078tdx/Tm/Vyf3z31xe8bY7vrszpu7/fFTYaUvqHrE/qMVPLsknxeWb+VwWRXZvTszeVQWZ6d3dLo0pZSN9AXVFmJXSRkzF2/h1ZXbKavyc3GfrkwelUXf5ASnS4ssdufDrR5Pc+kqRLS5R6jt+44xfVEeb64uxG8MVw7ozj3ZmWR1ae90aZHH7ny41eNpLl2FkF6WiTDf7T7M9IV5zP96B1EiXDckMPcltWMbp0uLXHbnw60eT3Ppqgn0sozHrC8sYWru93z8zW7atI7i9uHp/PL8DE7tEOd0aZHP7ny41eNpLl2FkDZ3l/siPzD35fPviukQF819o7O4bXhPTmnb2unSvMPufLjV42kuXYWQC8POyhjDws17uO7pZVz/zHK+KSrhoYt7s/Th0Tzw897a2O1mdz7c6vE0l65CSM/cXaS62vDJxl3k5OaxvqiEbglx/PHyM7nh7DTiW7tkRIAX2Z0Pt3o8zaWrEGrwBVURSQVeBLoC1cAMY8yTQftkA+8A+TWb3jLGTKnvuPqC6o+q/NXM/3oH0xbm4dtzhPSkNtydncnVA1NoHa2/XCmlfmTnC6pVwG+MMWtEpD2wWkQ+NcZsDNpvsTHmsqYU21KVV/mZu7qQpxflUbC/lNO7tuepCQO5tF83olq5/N2kjeX2PLfm0ptHHxfXabC5G2N2Ajtrvj8sIpuAZCC4uSuLjlVU8erK7cxcvIXdh8o5KzWR31/WhwtO70IrrzV1cH+eW3PpzaOPiys1KucuIunA50BfY8yhE7ZnA28ChcAO4LfGmG/qO1ZLvCxTUlrJi8u28uzSfA4cq2RYRhL3js7iZ5lJ7p/70hxuz3NrLr159HEJK9tz7iLSjkAD/9WJjb3GGqCHMeaIiIwF5gG9ajnGJGASQFpamtW7jnh7j5Qze0k+Ly3fxpHyKkaf3oXJo7IY3OMUp0sLD7fnuTXico0AAAALD0lEQVSX3jz6uLiSpeYuIjEEGvsrxpi3gm8/sdkbYz4QkWki0skYszdovxnADAicuTer8giw42ApMz7fwmtfbqe8qpqx/bpxT3Ymfbq3sLkvbs9zay69efRxcaUGoxgSuF4wG9hkjKl1FqmIdK3ZDxE5p+a4++wsNJJs3XuU381dx8gncnl5xTYu69+dfz0wkpwbB7W8xg7uz3NrLr159HFxJStn7sOBW4D1IrK2Ztt/AGkAxpingXHA3SJSBZQC441TQ2sctHnXYXJyfby3bgfRUa2YcE4ak0ZkkHJKC5/74vY8t+bSm0cfF1fSwWE2WFtwkJxcH59u3E3b1lHcPKwHd5zXky7tde6LUspeOjgsxIwxrNiyn5xcH0t8e0mIj+FXF/Zi4s/SSWzjgfEALS23/N4D9n0iklIuoM29kYwx5G7eQ05uHqu3HaBTu1geueR0bjq3B+1iPfJwtrTc8nsPwKrZP/5s/D/+rA1eRSiPdKPQ81cbPtqwi5xcHxt3HiI5MZ4/XdmH64akEhfjsbkvn035sbEfV1ka2O7F5r76+bq3a3NXEUqbewMq/dXM+6qI6Yvy2FJ8lIxObXliXH+uGphMTJRH5760tNyy8Tduu1IRQJt7Hcoq/fxzVQFPL9pC0cFSzujWgZwbB3Fx367em/sSrKXlliWq9kYuHvuNTLUo2tyDHCmv4tWV25i5OJ/iw+UMSkvkT1f1YVTvLt4eEXCiC35/8jV38HZuefDEk6+5n7hdqQilzb3GwWMVPL9sK88t3UpJaSXnZXXiyfEDGJbh8bkvtWlpueXj19U1LaM8pMXn3PccLmP2knxeXr6NoxV+LjzjVCaPymRgWguZ+6KUiiiac29A0cFSnlmUx+tfFlDpr+ay/t25Z1Qmp3ftEJb7n/dVEU98vJkdB0vpnhjPg2N6c9XA5LDct628kof3yjqcoo+f67S45r6l+AjTF+bx9ldFiMA1A1O4KzuTnp3ahq2GeV8V8chb6ymtDLyIV3SwlEfeWg8QWQ3eK3l4r6zDKfr4uVKLuSyzccchchb6+GD9TmKjWzH+7MDcl+6J8Q3/ZZsNf2wBRQdLf7I9OTGepQ+PDns9TeaVOd5eWYdT9PELK70sU2PN9gPkLPDx2bd7aBcbzV0jM7njvJ50ahfrWE07amns9W13La/k4b2yDqfo4+dKnmzuxhiW5e1j6gIfy7fsI7FNDA9cdBq/GJZOQpsYp8uje2J8rWfuTvwW0SxeycN7ZR1O0cfPlTz1FktjDP/auJurpy3jplkrySs+wqOXnsHS343mvgt6uaKxAzw4pjfxQSML4mOieHBMb4cqaiKvzPH2yjqcoo+fK3nizN1fbXh//U6m5fr4dtdhUjvG8+er+3LtoBRXzn05/qJpxKdlvJKH98o6nKKPnytF9AuqFVU/zn3J33uUrC7tuCc7kyvO6k60V+e+KKVaNE+/oFpW6ee1L7Yz4/Mt7Cgpo0/3Dky/aRBj+nSlldfnvihVH7vz5ppfj1gR19xzv93Dg3O/Zu+RCs5OP4W/XNOPkad1bnkjApQKZnfeXPPrES3imntaUhvO7J7A5OxMhmYkOV2OUu5h9xz+ljbX32Mirrlndm7Hi7ef43QZSrmP3Xlzza9HNH3VUSmvqCtX3tS8ud3HU2GlzV0pr7A7b6759YimzV0pr+h/PVz+VGCmCxL4evlTTb8+bvfxVFhFdM5dKaVaGqs5dz1zV0opD9LmrpRSHqTNXSmlPEibu1JKeZA2d6WU8iBt7kop5UHa3JVSyoO0uSullAc12NxFJFVEckVkk4h8IyL317KPiMhTIuITkXUiMig05SrXWfcG/L0v/DEx8HXdG05XpJTC2lTIKuA3xpg1ItIeWC0inxpjNp6wzyVAr5o/Q4HpNV+Vl+m8b6Vcq8Ezd2PMTmPMmprvDwObgOAP+7wSeNEErAASRaSb7dUqd6lv3rdSylGNuuYuIunAQGBl0E3JQMEJPxfy038AEJFJIrJKRFYVFxc3rlLlPjrvWynXstzcRaQd8CbwK2PMoeCba/krP5lIZoyZYYwZYowZ0rlz58ZVqtxH530r5VqWmruIxBBo7K8YY96qZZdCIPWEn1OAHc0vT7mazvtWyrWspGUEmA1sMsb8Tx27zQdurUnNnAuUGGN22linciOd962Ua1lJywwHbgHWi8jamm3/AaQBGGOeBj4AxgI+4Bhwm/2lKlfqf702c6VcqMHmboxZQu3X1E/cxwCT7SpKKaVU8+g7VJVSyoO0uSullAdpc1dKKQ/S5q6UUh6kzV0ppTxIm7tSSnmQNnellPIgCUTUHbhjkWJgWxP/eidgr43lOMkra9F1uItX1gHeWYtd6+hhjGlwOJdjzb05RGSVMWaI03XYwStr0XW4i1fWAd5ZS7jXoZdllFLKg7S5K6WUB0Vqc5/hdAE28spadB3u4pV1gHfWEtZ1ROQ1d6WUUvWL1DN3pZRS9XB9cxeRKBH5SkTeq+W2WBF5XUR8IrKy5jNeXamBdUwUkWIRWVvz55dO1GiFiGwVkfU1da6q5XYRkadqnpN1IjLIiTobYmEd2SJScsJz4sqPlxKRRBGZKyLfisgmERkWdHukPB8NrSNSno/eJ9S4VkQOicivgvYJy3Ni5cM6nHY/sAnoUMttdwAHjDFZIjIeeBy4IZzFNUJ96wB43RhzbxjraY5Rxpi68rqXAL1q/gwFptd8daP61gGw2BhzWdiqaZongY+MMeNEpDXQJuj2SHk+GloHRMDzYYzZDAyAwAkdUAS8HbRbWJ4TV5+5i0gKcCkwq45drgReqPl+LnBBzccCuoqFdXjJlcCLJmAFkCgi3ZwuyotEpAMwgsDHYGKMqTDGHAzazfXPh8V1RKILgDxjTPCbNcPynLi6uQP/AB4Cquu4PRkoADDGVAElQFJ4SmuUhtYBcG3Nr2hzRSS1nv2cZoBPRGS1iEyq5fYfnpMahTXb3KahdQAME5GvReRDEekTzuIsygCKgedqLvnNEpG2QftEwvNhZR3g/ucj2HhgTi3bw/KcuLa5i8hlwB5jzOr6dqtlm6viPxbX8S6QbozpD/yLH38bcaPhxphBBH61nCwiI4Jud/1zUqOhdawh8Dbvs4D/BeaFu0ALooFBwHRjzEDgKPBw0D6R8HxYWUckPB8/qLm0dAXwz9purmWb7c+Ja5s7gQ/mvkJEtgKvAaNF5OWgfQqBVAARiQYSgP3hLNKCBtdhjNlnjCmv+XEmMDi8JVpnjNlR83UPgWuJ5wTt8sNzUiMF2BGe6qxraB3GmEPGmCM1338AxIhIp7AXWr9CoNAYs7Lm57kEmmTwPm5/PhpcR4Q8Hye6BFhjjNldy21heU5c29yNMY8YY1KMMekEfr1ZYIy5OWi3+cAvar4fV7OPq85KrKwj6HrbFQReeHUdEWkrIu2Pfw/8HNgQtNt84NaaRMC5QIkxZmeYS62XlXWISNfjr9+IyDkE/l/ZF+5a62OM2QUUiEjvmk0XABuDdnP982FlHZHwfASZQO2XZCBMz0kkpGVOIiJTgFXGmPkEXoB5SUR8BM7YxztaXCMEreM+EbkCqCKwjolO1laPU4G3a/4fiwZeNcZ8JCJ3ARhjngY+AMYCPuAYcJtDtdbHyjrGAXeLSBVQCox324lDjX8HXqm5DLAFuC0Cnw9oeB2R8nwgIm2Ai4B/O2Fb2J8TfYeqUkp5kGsvyyillGo6be5KKeVB2tyVUsqDtLkrpZQHaXNXSikP0uaulFIepM1dKaU8SJu7Ukp50P8Hc4IvqJsTdqsAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "x_ponits = np.arange(4, 8)\n",
    "# x1 取 x_ponits 时， x2 的值\n",
    "y_ = -(lr_clf.weights[1]*x_ponits + lr_clf.weights[0])/lr_clf.weights[2]\n",
    "plt.plot(x_ponits, y_)\n",
    "\n",
    "#lr_clf.show_graph()\n",
    "plt.scatter(X[:50,0],X[:50,1], label='0')\n",
    "plt.scatter(X[50:,0],X[50:,1], label='1')\n",
    "plt.legend()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": true,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
